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ABSTRACT 


The  Quantification  of  Effectiveness  (QUEF)  program 
was  initiated  to  develop  a  simplified  method  of  measuring 
systems  ef fectivenesa  and  to  determine  the  sensitivity  of 
systems  effectiveness  to  logistics  factors.  The  initial 
approach  was  to  quantify  systems  effectiveness  from  the 
equipments'  mission  essential  hardware  characteristics. 
This  approach  was  discontinued  because  the  hardware 
characteristics  identified  were  too  detailed  to  s'*  low  a 
general  method  to  be  developed.  The  second  approach 
Involved  the  identification  of  the  logistics  factors  that 
influence  systems  effectiveness  and  the  reallocation 
of  resources  among  the  most  sensitive  of  the  logistics 
factors.  This  report  describes  the  methodologies  employed 
in  the  two  approaches,  discusses  the  problem,  and  presents 
the  results. 


ADMINISTRATIVE  INFORMATION 


The  Quantification  of  Effectiveness  (QUEF)  project  was  initiated  by  the 
Naval  Air  Systems  Command  (NAVAIR  340C).  Funding  was  provided  under  AIRTASK 
Number  A3A00O00/01OB/1F60532O0O.  The  David  Taylor  Naval  Ship  Research  and 
Development  Center  (DTNSRDC)  undertook  the  project  in  FY-80;  the  Logistics 
Divisior  (Code  1870)  of  the  Computation,  Mathematics  and  Logistics  Department 
was  the  performing  organization. 
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GECTION  1 
INTRODUCTION 


The  Quantification  of  Effectivenaa  (QUEF)  program  was  initiated  in  FY-80 
with  the  broad  objective  of  developing  a  simplified  approach  to  quantifying  sys¬ 
tems  effectiveness  and  performing  trade-off  analyses  of  logistics  support  alterna¬ 
tives.  Current  methods  of  quantifying  systems  effectiveness  are  very  complex 
and  cumbersome,  Involving  detailed  system  descriptions  and  large  amounts  of 
data.  The  complexity  of  the  approach  often  makes  people  reluctant  to  perform 
the  analysis.  The  NAVAIR  sponsor  felt  that,  if  a  simplified  method  could  be 
developed  which  would  allow  systems  effectiveness  to  be  determined  without  the 
complexity  of  the  present  approach,  equipment  developers  and  managers  would  be  more 
likely  to  perform  a  systems  effectiveness  analysis.  For  this  reason  the  Initial 
approach  considered  quantifying  systems  effectiveness  by  the  mission  essential 
hardware  characteristics  of  the  equipment. 

In  today's  environment  of  tight  budget  control,  the  operating  and  support 
costs  associated  with  weapons  systems  are  coming  under  closer  examination.  The 
logistics  support  associated  with  a  weapons  system  greatly  affects  the  systems 
effectiveness  and  cost  effectiveness  of  equipment.  The  second  approach  con¬ 
sidered  for  the  QUEF  project  was  to  identify  the  logistics  factors  which  affect 
systems  effectiveness  and  to  perform  trade-off  analyses  and  sensitivity  studies  of 
the^’e  logistics  factors  for  selected  equipment.  In  this  way  program  managers  would 
be  able  to  determine  early  in  development  which  logistics  factors  drive  the  operating 
and  support  costs,  and  managers  of  operational  systems  would  be  able  to  make  logical 
decisions  regarding  reallocation  of  logistics  support  resources. 

This  report  presents  a  brief  description  of  systems  effectiveness,  including 
the  availability,  dependability,  and  capability  functions,  and  discusses  the 
effectiveness  disciplines  (reliability,  maintainability,  and  logistics  support) 
which  contribute  to  systems  effectiveness  technology.  It  describes  the  methodol¬ 
ogies  employed  in  the  two  approaches,  discusses  the  problem,  and  presents  the 
results. 
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SECTION  2 

SYSTEMS  EFFF  TIVENESS 


During  thp  acquisition  of  a  new  weapons  system,  comparison  of  the  effective¬ 
ness  of  the  various  configurations  per  unit  cost  gives  a  relative  ranking  of  the 
values  of  the  configurations.  The  cost  of  the  system  includes  both  acquisition  and 
operation  and  support  costs.  The  need  to  calculate  both  the  effectiveness  and  the 
cost  of  the  system  begins  during  the  conceptual  phase  and  continues  throughout  the 
life  of  the  system. 

Systems  and  cost  effectiveness  can  be  u'Jed  together  as  a  decision-making  tool 
for  management.  The  systems  and  cost  effectiveness  procedures  permit  many  system 
designs  and  use  criteria,  such  as  mission  requirements,  technical  performance 
parameters,  design  factors,  and  resource  allocations,  to  be  integrated  and 
evaluated  against  similar  criteria  for  other  systems/configurations.  The  use  of 
systems  and  cost  effectiveness  during  a  system's  life  cycle  provides  management 
with  information  to  be  used  in  making  Important  design  and  operating  decisions. 

Systems  effectiveness  is  defined  as  the  probability  that  a  system  can 
successfully  meet  an  operational  demand  throughout  a  given  period  when  operated 
under  specified  conditions.  An  alternative  definition  is  that  systems  effective¬ 
ness  is  a  measure  of  the  degree  to  which  an  item  can  be  expected  to  meet  a  set  of 
specific  mission  requirements,  and  may  be  expressed  as  a  function  of  availability, 

dependability,  and  capability.  For  a  detailed  discussion  of  systems  effectiveness 

1*  2 

the  reader  is  referred  to  Hanifon  and  Chop  ,  which  provide  excellent 
background  material. 

To  evaluate  systems  ef fectivenss,  a  Figure  of  Merit  (FOM)  must  often  be 
defined.  An  FOM  is  a  measure  of  systems  effectiveness  related  to  one  or  more  of 
the  missions  the  system  will  be  required  to  perform.  Different  types  of  aircraft 
might  have  the  following  typical  FOM's; 

o  Transport  Aircraft  -  expected  number  of  ton  miles  per  unit  of  time 
o  Tactical  Aircraft  -  probability  of  providing  timely  troop  support  over 

X  miles  with  y  sorties 


*  A  complete  listing  of  references  is  given  on  page  27. 
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o  Interceptor  /tlrcraft  -  probability  of  destroying  n  out  of  m  enemy  air¬ 
craft  of  a  specific  type  per  engagement. 

FOM'a  must  be  defined  very  carefully  so  as  not  to  bias  the  results  of  the  analysis. 
2.1  AVAILABILITY 

Availability  is  defined  as  the  degree  to  which  an  item  is  operable  and 

commlttable  at  the  start  of  a  mission,  when  the  mission  is  called  for  at  an  unknown 

(random)  time.  Another  definition  is  the  probability  that  the  system  is  in  an  "up" 

and  ready  state  at  the  beginning  of  the  mission  wnen  the  mission  occurs  at  a  random 

time;  i.e.,  the  system  is  ready  to  operate  within  allowable  response  time  with  all 

mission-required  functions  capable  of  operating  within  design  specifications. 

Inherent  (or  ideal)  availability  is  expressed  in  terms  of  MTBF  (mean  time 

between  failures)  and  MTTR  (mean  time  to  repair): 

A  -  MTBF 

^  MTBF  +  MTTR 

Inherent  availability  assumes  only  active  components  of  corrective  maintenance; 

i.e.,  no  waiting  for  spares  and  technicians,  no  detection  or  administrative  time, 

no  downtime  due  to  preventive  maintenance  or  servicing,  immediate  availability  of 

technical  manuals,  test  equipment,  and  software,  etc. 

Achieved  availability  includes  preventive  maintenance: 

A  -  MTBM 

®  MTBM  +  MADT 

where  MTBM  is  mean  time  between  maintenance  events  (both  corrective  and  preventive) 

and  MADT  is  mean  active  downtime,  which  includes  the  active  (non-waiting)  time 

element  of  both  preventive  and  corrective  maintenance. 

Operational  availbillty  is  expressed  by  MTBM  and  MDT  (mean  downtime): 

A  ,  -  MTBM 

MTBM  +  MDT 

Mean  downtime  includes  all  non-operable  time,  active  and  inactive,  including 
"software"  downtime,  supply  and  administrative  delay  times,  and  corrective  and 
preventive  maintenance  times.  assumes  the  availability  of  an  operator  if  one 

is  required. 

Factors  influencing  availability  include  manning,  operations,  maintenance, 
and  logistics  support. 
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2.2.  DEPENDABILITY 

Dependability  is  defined  as  the  probability  that  an  available  system 
will  continue  to  operate  throughout  the  mission  either  without  a  system-level 
failure  or  if  the  system  fails,  with  restoration  to  operation  within  some  critical 
time  interval  which,  if  exceeded,  would  result  in  mission  failure.  The  various 
kinds  of  dependability  depend  on  the  criteria  for  system  failure,  and  each  kind 
requires  a  different  model.  The  following  examples  of  such  crlterif^  are  given  by 
Hanifan^; 

o  no  failure  allowable 

o  no  system-level  failures  allowable,  but  certain  element  failures  may 
occur  without  repair 

.  AO  t. ystem-level  failures  allowable,  but  certain  element  failures  may 
occur  with  repair  a  given  number  of  times 

o  sybtem -level  failures  allowable  if  downtime  is  less  than  a  specified 
tj.  -e 

Of  '  (  “iteria  also  exist. 

Di'pendabii ity  is  a  function  of  the  reliability  and  maintainability  charac- 
te.'l  '.t  l"  1  of  a  ys,..em.  The  references  describe  many  models  which  reflect 
differtiiv.  s'yarems,  missions,  and  criteria  for  failure. 

2.3  CAPABILITY 

Capability  is  the  probability  that  the  system's  designed  performance  will 
meet  mission  demands  successfully,  assuming  that  the  system  is  available  and 
dependable.  This  terra  takes  into  account  the  adequacy  of  the  system  elements  to 
carry  out  the  mission  when  operating  in  accordance  with  the  system  design  specifi¬ 
cations  as  affected  by  the  environment.  Both  machine  and  human  modules  of  the 
operable  system  are  Included. 

The  human  capability  term  is  the  probability  that  the  operator  will  respond 
successfully  to  mission  demands,  assuming  that  he  is  dependable  and  that  the  hard¬ 
ware  is  both  dependable  and  capable.  This  term  includes  human  error  and  reflects 
the  effects  on  operator  performance  of  training,  experience,  change  of  performance 
as  a  function  of  mission  stress  and  duration,  motivation  forces,  etc. 
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Hardware  capability  is  the  probability  that  the  hardware  will  successfully 
meet  mission  demands,  assuming  it  is  available  and  dependable,  and  assuming  the 
operator  is  available,  dependable,  and  capable.  Overall  system  capability  is 
the  product  of  human  capability  and  hardware  capability. 

2.4  EFFECTIVENESS  DISCIPMNES 

Some  of  the  factors  discussed  by  Hauifan  ^which  affect  systems  effec- 
tivenefjs  are  shown  in  Figure  1.  Availability  depends  on  both  operator  avail¬ 
ability  and  hardware  availability.  Dependability  and  capability  also  have 
operator  and  hardware  components.  Hardware  availability  and  hardware  dependa¬ 
bility  are  generally  affected  by  the  same  factors,  the  ef feCi,iveness  disciplines 
some  of  which  are  discussed  in  the  following  paragraphs. 

2.4.1  Reliability 

Reliability  is  defined  as  the  probability  that  an  item  will  periorm 
its  intended  function  for  a  specified  interval  under  stated  conditions. 
Reliability  is  a  well-established  effectiveness  discipline,  with  program 
activities  structured  to  provide  an  item  with  failure  characteristics 
compatible  with  system  availability  and  dependability  requirements.  The  key 
parameter  from  a  reliability  analysis  is  MTBF,  which  relates  directly  to 
availability  and  dependability. 

If  predicted  reliability  is  unsatisfactory,  redundancy,  parts  selection 
and  screening,  derating,  cooling,  and  special  designs  are  some  of  the  cost-effec 
tive  methods  used  to  achieve  the  desired  reliability  characteristics.  In 
designing  an  item  to  maximize  inherent  reliability,  particular  attention  is  paid 
to  parts  selections,  control  and  screening,  temperature  and  stress  derating, 
environmental  control,  redundancy,  and  design  simplifications. 

2.3.2  Maintainability 

Maintainability  is  a  characteristic  of  design  and  Installation  which 
is  expressed  as  the  probability  that  an  item  will  be  retained  in  or  restored 
to  a  specified  condition  within  a  given  period  of  time,  when  the  maintenance 
is  performed  in  accordance  with  prescribed  procedures  and  resources.  Maintain¬ 
ability  is  also  a  well  established  effectiveness  discipline,  with  the  key 
parameter,  MTTR,  being  directly  related  to  availability  and  dependability. 
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SYSTEM  EFFECTIVENESS 


Maintainability  analysis  ensures  that  a  system's  maintainability 
characteristics  are  consistent  with  systems  effectiveness  goals.  Typical 
tra'ie- of  fs  performed  include  repair  vs.  discard  of  failed  modules,  loca¬ 
tion  of  module  or  equipment  repair  (shipboard,  tender,  depot,  contractor, 
etc.),  test  point  location,  type  and  location  of  failure  detection  devices 
and  test  equipment  (manual  vs.  automatic,  special  vs.  general,  built-in 
vs.  portable),  technician  numbers  and  skill  levels,  special  support  equipment, 
tools  and  jigs,  location  and  types  of  maintenance  data  and  instructions, 
and  level  of  modulization  and  degree  of  standardization. 

In  order  to  improve  maintainability,  particular  attention  is  paid  to 
fault  isolation  times,  skill  requirements,  speed  of  replacement,  inter¬ 
changeability,  and  repair  quality. 

2. A. 3  Logistics  Support 

Shortages  of  spares  and  delays  in  obtaining  spares,  technicians >  tools, 
support  equipment,  data,  and  other  supporting  elements  are  responsible  for  most 
system  downtime  in  many  operational  systems.  Since  mean  downtime 
(MDT)  is  a  contributing  term  to  operational  availability  and  dependability,  it 
can  be  seen  that  logistics  support  is  a  major  factor  in  systems  effectiveness. 
The  effectiveness  of  many  systems  is  highly  sensitive  to  the  design  of  the 
logistics  support  systems,  as  is  the  life  cycle  cost.  For  this  reason  the 
Integrated  Logistic  Support  (ILS)  system  is  being  used  to  provide  effective 
support  throughout  the  life  cycle  of  a  system.  The  ILS  function  is  supported 
by  the  systems  effectiveness  analysis  function,  which  provides  the  essential 
analytic  framework  for  making  rational  logistics  support  decisions  within  an 
overall  operational  context.  Some  of  the  logistics  support  costs  which  are 
examined  are  initial  and  pipeline  spares  cost,  replacement  spares  cost, 
on-equipment  maintenance  cost,  inventory  entry  and  supply  management  cost, 
support  equipment  cost,  cost  of  personnel  training  and  training  equipment, 
and  cost  of  management  and  technical  data. 
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SECTION  3 

HAiiDWARE  CHARACTERISTICS  APPROACH 


3.1  BACKGROUND 

The  initial  systems  effectiveness  project  for  wV ich  DTNSRDC  was  tasked 
involved  the  prediction  of  systems  effectiveness  as  a  function  of  the  equipment's 
hardware  characteristics.  During  the  acquisition  of  a  weapons  system,  a  continuous 
requirement  exists  to  evaluate  trade-offs  between  cost  and  systems  effectiveness 
of  many  alternatives  associated  with  design  and  logistics  support.  Coats  can  be 
quantified,  but  systems  effectiveness  is  obtainable  only  through  subjective  eval¬ 
uation.  Currently  there  is  no  consistent  method  for  the  evaluation  of  systems 
effectiveness.  Any  evaluation  that  is  accomplished  is  done  subjectively  with  no 
guarantee  that  all  important  factors  of  systems  effectiveness  are  considered  or 
that  the  factors  are  given  consistent  weights.  In  the  past,  efforts  at  quantify¬ 
ing  systems  effectiveness  have  ranged  from  Intuitive  thinking  to  extremely 
sophisticated  mathematical  models  requiring  large  amounts  of  time  and  information 
for  implementation.  Moat  of  these  efforts  lie  at  one  end  of  this  spectrum  and 
are  difficult  or  impossible  to  accomplish  because  of  the  v’ork  load  generated. 

For  these  reasons  it  was  decided  to  attempt  to  develop  a  method  to  quantify 
systems  effectiveness  on  the  basis  of  hardware  characteristics. 

3.2  METHODOLOGY 


3.2.1  Objective 

The  objective  was  to  determine  the  feasibility  of  quantifying  systems 
effectiveness  and  of  performing  trade-off  analyses  of  logistics  support  alterna¬ 
tives  by  identifying  mission  essential  hardware  characteristics  and  by  establish¬ 
ing  the  probability  that  these  characteristics  will  be  operable  in  a  combat 
environment . 

3.2.2.  Approach 

Perform  the  project  as  follows: 

1.  Conduct  a  literature  search  to  acquire  the  necessary  information  on 
systems  effectiveness. 
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2.  Identify  mission  essential  hardware  characterisitcs . 

3.  Determine  a  weighting  factor  for  each  mission  essential  hardware 
characteristic. 

4.  Develop  a  method  of  calculating  the  probability  that  each  mission 
essential  hardware  characteristic  will  be  operable  in  a  combat  environment. 

5.  Select  representative  equipment  to  demonstrate  the  method  and  to 
perform  trade-off  analyses. 

3.3  DISCUSSION 

3.3.1  Literature  Search 

Literature  searches  were  conducted  of  the  data  bases  maintained  by  the 
Defense  Logistics  Systems  Information  Exchange  (DLSIE)  and  the  Defense  Technical 
Information  Center  (DTTC).  The  following  keywords  were  used  to  search  the 
data  bases: 

o  Systems  effectiveness 
0  cost  effectiveness 
0  maintainability 
0  reliability  prediction 
o  design  factors 
0  system  design  parameters 
0  hardware  characteristics 

Tlie  bibliographies  i. rovlded  by  these  two  services  contained  many  reports  with 
general  information  in  the  area  of  systems  effectiveness  and  many  reports  of  very 
detailed  work  in  specific  areas,  but  no  reports  on  work  similar  to  the  present 
task.  Many  reports  were  ordered,  the  most  important  being  the  onus  used  as 
references  for  this  report. 

3.3.2  Hardware  Characteristics 

From  the  review  of  many  reports  it  is  possible  to  make  some  generalized 
statements.  First,  equipment  has  physical  characteristics,  such  as 
o  weight 
0  volume 
o  shape 
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o  energy  levels 

o  mechanical  and  electrical  packaging 
o  environmental  ''apabilltles 

Second,  equipment  has  performance  characteristics,  s 

o  accuracy 
u  speed 
o  range 
o  capacity 
o  power  output 
u  discriminations 

Tlic  following  additional  hardware  characteristics  were  encountered  in  various 
reports: 

o  englneuvlng  design 
o  complexity 

o  numbei  and  accessibility  of  test  points 
u  standatd/non-staudard  parts 
0  modules 

o  malntentdice  philosophy 
0  number  of  components 
o  stress  levels 
0  spacu/environment  conditions 
o  redundancy 
0  failure  rate 
o  repair  time 

Obviously,  many  different  hardware  churac Ccr Isticu  can  be  used  to  describe 
a  weapons  systems,  some  very  general,  some  very  Hi)Mcific.  Many  apply  to  all 
types  of  equipment,  but  some  apply  only  to  certain  types  of  equipment.  The  task 
of  compiling  a  list  of  all  hnruwure  chaructcristlcH  that  apply  to  all  systems  is 
formidable. 

No  attempt  was  made  in  this  phase  of  the  project  to  determine  which  hard¬ 
ware  characteristics  are  essential  to  all  missions.  Analysis  of  data  from  many 
types  of  aysteroa  would  be  required  to  make  tills  determination. 
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3t3.3  Weighting  FactTrs 

T.£  hardware  characterlatics  are  selected  which  apply  to  all  Navy  equipment, 
it  is  apparent  that  certain  characteristics  will  be  more  important  to  some 
equlpraont  than  to  others.  Therefore,  weighting  factors  are  needed  to  apply  the 
hardware  characteristic  in  the  proper  degree  to  each  type  of  equipment.  The 
sclieine  envisionod  is  shown  in  Table  1.  The  hardware  characteristics  are  the  same 
tor  all  types  of  equipment  and  for  this  example  are  numbered  1  to  n.  Each  type 
of  equipment  (c.g.,  radio,  radar,  computer)  will  have  a  weighting  factor  associ¬ 
ated  with  each  characteristic.  The  weighting  factors  will  have  the  same  value 
within  an  equipment  type,  but  the  value  of  the  weighting  factor  can  vary  among 
types.  I'or  example,  the  value  of  the  weighting  factor  for  characteristic  1  for 
radios  may  be  10,  but  for  radars  it  may  be  S.  Characteristic  2  may  be  valued  at 
^ero  fur  radios,  but  20  for  radars.  In  this  way  the  values  of  the  weighting 
luclota  fur  tlie  characteristics  may  be  tailored  for  each  typo  of  equipment.  The 
wuigl\tlug  factors  for  each  tvpe  will  sum  to  100. 

tills  approach,  all  types  of  equipment  will  have  the  same  hardware 
I'haracterisL iCH ,  but  the  impottuuce  of  the  characteristics  will  vary  with  the 
type  of  oquL|)muut. 
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TABLE  1  -  SYSTEMS  EFFECTIVENESS  HARDWARE  CHARACTERISTICS  WEIGHTING  FACTORS 


Probabilltltia 

Thu  touk  of  calculating  the  probability  of  each  inlsalon  uuauntlul  hardware 
charucterlatlc  being  operable  under  combat  environment  would  beat  be  done  using 
operational  data  for  the  equipment  being  unalyi^ed.  However»  early  In  development 
theue  dutu  are  not  avullablct  Other  sourcew  of  these  data  would  bo  design  data, 
test  dutu,  operational  dota  of  similar  equipment,  or  expert  opinion  from 
knowlodgable  people. 

3.3.5  Demonstration 

Once  the  essential  hardware  churucterlstlcs  are  Identified  and  the  weight¬ 
ing  fuctors  find  probabilities  determined,  representative  equipment  should  bo 
selected  to  demonstrate  the  method.  Several  ulternatlvea  should  be  developed 
for  the  chosen  equipment,  und  the  systems  effectiveness  calculated  us  shown  In 
Table  2.  In  this  way  the  relative  worth  of  each  alternative  cun  be  examined. 


TABLE  2  -  SYSTEMS  EFFECTIVENESS  HARDWARE  CHARACTERISTICS  QUANTIFICATION 


HARDWARE  CHARACTERISTICS 


3.4.  RESULTS 

Examlnntion  o£  the  literature  provided  the  background  Information  on  systems 
effectiveness  and  many  of  the  hardware  characteristics  Lhat  should  be  considered. 
The  next  steps  involved  determining  the  mission  essential  hardware  characteristics 
and  the  weighting  factors  for  each.  However,  because  of  the  close  association 
between  reliability  and  systems  effectiveness,  it  was  decided  to  first  investigate 
the  prediction  of  reliability  characteristics  and  to  determine  whether  the 
chararacteristlcs  used  to  predict  reliability  could  also  be  used  to  predict 
systems  ef foctiveness. 

3 

A  report  from  the  Army  Material  Command  describes  reliability  prediction 
techniques  and  compares  the  analyses  perforated  by  each  method.  A  report  published 

A 

by  the  Naval  Air  Development  Center  presents  a  method  for  predicting  relia¬ 
bility  of  non-nvlonics  equipment  using  existing  field  data  and  a  reliability 
technology  improvement  factor.  Most  non-avionics  subsystems  of  the  aircraft  were 
considered  in  this  report.  Of  significance  to  the  systems  effectiveness  analysis 
was  a  list  of  aircraft  characteristics  (Including  engine)  and  operational  charac¬ 
teristics  used  in  the  formulation  of  the  reliability  prediction  equations. 

A  report  published  by  the  Hughes  Aircraft  Company  presented  results 
of  n  study  to  develop  a  reliability  prediction  technique  to  estimate  system 
complexity  based  on  system  performance  data  derived  from  design  specifications, 
detailed  parts  auramarles,  and  detailed  handbook  predictions.  Four  major  types  of 
equipment  were  considered:  radars,  computers,  displays,  and  communications.  The 
design  parameters  used  to  predict  reliability  for  each  type  of  equipment  are  listed 
in  Table  3.  The  only  characteristic  which  appears  for  every  type  of  equipment  is 
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TABLE  3  -  EQniP>lE^;T  CHARACTERISTICS  FOR  FOUR  TYPES  OF  EQUIPMENT 


design  year.  All  other  characteristics  are  specifically  related  to  the  type  of 
equipment.  Since  reliability  is  closely  associated  with  systems  ef fectivenebs, 
it  is  believed  that  the  lack  of  correspondence  in  detailed  hardware  characteris¬ 
tics  found  in  reliability  prediction  will  also  be  found  in  the  prediction  of 
systems  ef lectlveness.  Although  it  might  be  possible  to  determine  hardware 
characteristics  which  will  enable  systems  effectiveness  to  be  predicted  for 
spectflc  equipment,  broad  general  hardware  characteristics  which  could  be  used 
to  prcnlict  systems  effectiveness  for  a  wide  range  of  equipments  could  not  be 
developed.  Consultation  with  sponsors  and  supervisory  personnel  led  to  the 
decision  to  abandon  tliis  approach  to  systems  effectiveness  analysis  at  least  for 
tlie  present. 


If) 


SECTION  4 

LOGISTICS  FACTORS  APPROACH 


4 . 1  BACKGROUND 

The  next  approach  was  to  study  the  feasibility  of  quantifying  the  changes  in 
systems  effectiveness  resulting  from  reallocation  of  logistics  support  resources* 
The  initial  attempt  to  develop  measures  of  effectiveness  (MOE's)  that  reflect  the 
logistics  aspects  of  systems  effectiveness  was  refocused  as  the  program  progressed 
on  the  logistics  factors  that  influence  systems  effectiveness. 

During  the  design  and  development  of  a  weapons  system  many  trade-offs  are 
made  between  the  capabilities  of  a  system,  the  logistics  support  requirements 
of  the  system,  and  the  cost  of  the  system,  both  initial  investment  cost  and 
operating  and  support  costa.  The  Integrated  Logistic  Support  (ILS)  and  Life 
Cycle  Coat  (LCC)  concepts  are  used  during  the  acquisition  phase  to  provide  the 
system  with  the  most  effective  logistics  support  in  the  most  cost  effective 
manner,  consistent  with  mission  objectives. 

Tlie  effectiveness  of  a  system  can  be  measured  by  its  availability,  dependa¬ 
bility,  and  capability.  In  many  cases  these  functions  become  reliability, 
maintainability,  and  performance  characteristics.  Until  the  deployment  of 
a  weapons  system,  the  logistics  factors  which  influence  these  effectiveness 
functions  are  often  difficult  to  measure  or  predict  accurately.  Only  when  the 
system  is  operational  in  its  actual  environment  can  the  effect  and  cost  of  the 
logistics  factors  be  accurately  measured. 

However,  in  the  operational  phase  budgetary  considerations  often  make  the 
various  logistics  support  requirements  compete  for  available  funds.  Then  it  would 
be  useful  to  have  a  method  to  aid  equipment  managers  in  making  funding  decisions 
that  provide  maximum  effectiveness  at  minimum  cost  or  at  a  fixed  cost.  This 
program  attempts  to  provide  that  capability. 

4 . 2  METHODOLOGY 


4.2.1  Objective 

The  objective  was  to  develop  a  method  to  quantify  the  change  in  systems 
effectiveness  resulting  from  reallocation  of  logistics  support  resources. 


A. 2. 2  Approach 

The  project  Involved  the  following  steps: 

1.  Identify  logistics  factors  that  contribute  to  systems  effectiveness 

2.  Determine  how  the  logistics  factors  relate  to  the  systems  effective¬ 
ness  functions 

3.  Select  representative  sy8tem(s)  for  detailed  analysis 

A.  Determine  the  sensitivity  of  systems  effectiveness  to  the  logistics 
factors 

5.  Derive  cost  relationships  between  the  logistics  factors  and  systems 
effectiveness 

6.  Perform  sensitivity  studies  on  the  cost  of  logistics  factors  and 
systems  effectiveness 

7.  Develop  a  method  for  reallocating  resources 

8.  Apply  results  to  an  example  of  reallocation  of  logistics  resources 
A. 3  DISCUSSION 

A. 3.1  Logistics  Factors 

The  type  of  logistics  factors  that  should  be  examined  can  Initially 
be  selected  from  those  considered  in  the  ILS  elements.  The  ILS  elements  and 
representative  logistics  factors  are  as  follows: 

THE  MAINTENANCE  PLAN 

o  Maintenance  concept 

o  Reliability  and  maintainability  parameters  and  requirements 
o  Maintenance  tasks  (time  and  skill) 

0  Descriptions  of  maintenance  organizations 
o  Broad  support  and  test  equipment  requirements 
o  Maintenance  standards 
o  Broad  supply  support  requirements 
o  Facilities  requirements 
o  Replace/repair/discard  criteria 
o  Non-economic  engineering  evaluations 

-  item  size 

-  safety  requirements 
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-  technical  feasibility  of  repair 

-  support  and  teat  equipment  requirements 
o  Economic  cost  factors 

-  operations 

-  preventive  maintenance 

-  repair 

“  inventories 

-  documentation 

-  disposal 

o  Scheduled  maintenance  concepts 
o  Significant  items 
o  Remove/ replace  level 
o  False  alarm  rate 
o  Task  time 

o  Fault  Isolation  technology 
o  Minimum  maintenance  requirements 
o  Condition  monitoring 
o  Deferrable  maintenance  policy 
0  Software  requirements 
o  Modularity 

o  Unscheduled  maintenance 
o  Logistics  delay  time 

MANPOWER  PERSONNEL 

o  Personnel  quantities  needed 
o  Skill  levels 
o  Skill  specification 
o  Manhours  authorized 
o  Manhours  available 
o  Percent  utilization/productivity 
o  Manhours  per  flight  hour 
o  Cross-training 

o  Navy  Enlisted  Clsassifications  (NEC)/MOS 
o  Accession/attrition  rates 
o  Civilian  skills 
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SUPPLY  SUPPORT  (Including  Initial  provisloniriR) 
o  Spare  and  repair  parts  provisioning 
o  Consumption  and  usage  rates 
o  Recommended  allowances 
o  Supply  storage  requirements 
o  Operations  consumable  supply  requirements 

-  fuel 

-  lubricants 

-  oxygen 

o  Requirements  for  each  system  for 

-  storage  space 

-  supply  facilities 

-  equipment 

-  personnel 

-  procedures 

o  Rotable  pool  factor 
o  IMA  turnaround 
o  Resupply  time 
o  Number  of  items  required 
0  Quantity  per  use 
o  Mission  essentiality  code 
o  Weight 
o  Cube 

0  IMA/depot  pipeline  quantities 
o  Number  of  end  items  supported 
o  Initial  stockage  quantity 
c  Stock  levels 
o  Number  of  demands 

o  Number  of  items  repaired  IMA 

o  Number  of  items  repaired  depot 

o  Average/maxlmura  IMA  repair  times 
o  Number  of  backorder  days 

o  Number  of  units  demanded 

o  Percent  off-the-shelf 
o  Percent  demands  not  satisfied 
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o  Number  of  cannibalizatioiis 
o  Long  lead-time  items 
o  Retrograde  factor 


SUPPORT  AND  TEST  EQUIPMENT 

o  Equipment  identification 
o  Mainte'  nee  level 

o  Quantity  required  per  organization  per  location 
o  Equipment  function  and  capability 
o  Calibration  requirements 
o  Spares  and  repair  parts  listed 

o  Skill  levels  to  operate  and  maintain  equipment 
o  Average  utilization  (%) 

0  Average  length  of  time  used  (hours) 
o  Cost  per  unit 
o  Hours  authorized 
o  Hours  available 
o  %  used  -  scheduled  maintenance 
o  %  used  -  unscheduled  maintenance 
o  %  unused 

o  Number  of  units  demanded 
o  %  on  hand 

o  X  demands  not  satisfied 
o  Hours  backlog 

o  Length,  width,  height,  and  weight  of  unit 


TRAINING  AND  TRAINING  DEVICES 
o  Training  requirements 
o  Training  facilities 
o  Training  materials 
o  Types  of  cours  ■ 
o  Number  of  courses 
o  Duration  of  courses 
o  Number/skill  level  of  students 
o  Training  equipment  required 


o  Training  concept 
o  Training  aids 
o  Training  cycle 
o  Depot  training  factors 
o  Factory  training 

TECHNICAL  DATA 

o  System/ equipment  design 
o  Operations 
o  Maintenance 
o  Supply  support 
o  Types  of  manuals/data 
o  Contents  of  manuals/dat.a 
o  Publication  concepts 
o  Procedures 
o  Philosophy 
o  Scheduling 
o  Implementation 
0  Number  of  manuals 
o  Standardization 
o  Types  of  media 
o  Update/revision  methodology 

COMPUTER  RESOURCES  SUPPORT 

o  Computer  resources  support  reqireirents 

o  Computer  resources  support  facilities 

o  Computer  resources  support  materials 

o  Computer  resources  support  manpower 

Packaging,  Handling,  Storage,  and  Transportation 
o  Equipment  physical  dimensions 
o  Container  requirements  and  codes 
o  Storage  and  storage  space 
o  Preservation  and  packaging  requirements 
o  Handling  constraints 
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o 


Time 
0  Distance 
o  Mode 
o  Speed 
o  Environment 
o  Frequency 
o  Protection 
o  Storage  area 

FACILITIES 

o  Facilities  required  for  system  testing,  training,  operation,  and 
maintenance 
o  Requirements  for 

-  mobile,  portable,  and  air  transportable  vans 

-  mobile  maintenance  facilities 

-  shops 

-  training  facilities 

-  supply  storage 

-  bulk  storge  containers 

o  Facilities  identification  and  description 
o  Facility  design  criteria 
o  Facility  costs 
o  Lead  tirii'^s 

o  Facility  utilization  (%) 

o  Average  numbe  of  personnel  using  the  facility 
o  Maximum  number  of  personnel  using  the  facility 
o  Facility  space  dimensions 

0  Power  requirements  (electrical,  air,  hydraulic,  and  vacuum) 

0  Safety  requirements 
o  Storage  space 
o  Maintenance  communications 
0  Work  environment 
o  Servicing  -  Location/Quantity 
o  Composite  repair  facility  requirements 
o  Utilities  required 
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4.3.2  Systems  Effectiveness  Analysis 

Performing  a  systems  effectiveness  analysis  requires  an  analytical  tool 
with  which  to  perform  tlie  analysis,  usually  a  computer  program  which  models  the 
systems  effectiveness  characteristics  of  the  equipment  under  investigation.  The 
Handbook  of  Systems  Effectiveness  Models^  lists  many  models  which  are  used  in 
systems  effectiveness  analysis.  The  types  of  models  Include  cost,  ILS,  effective¬ 
ness,  availability,  reliability,  maintainability,  and  others.  Many  of  these 
models  were  developed  by  government  agencies  and  would  probably  be  available  for 
use  by  other  government  agencies.  Also,  since  this  handbook  was  compiled  in  1972, 
many  models  have  probably  been  written  since  tlie  handbook  was  published.  However, 
this  liandbook  is  still  a  uHotnl  starting  point  in  the  search  for  systems  effec¬ 
tiveness  models.  Other  sources  of  systems  effectiveness  models  would  probably  be 
the  SYSCUMS,  equipment  oriented  organizations,  and  contractors. 

Once  the  models  are  available  and  operational,  equlpmmont  to  be  analyzed 
must  be  selected.  This  is  beat  done  in  consultation  with  project  sponsors, 

SYHCDM  managers,  and  operational  organizations.  The  data  required  for  a  systems 
effectiveness  analysis  are  developed  througliout  the  life-cycle  of  u  weapons 
system.  Early  upproxlinat Ions  of  tlie  data  are  used  in  the  concept  formulation 
phase  and  are  rtsflned  and  imiu’oved  as  tho  aystem  undergoes  further  development. 

In  Clio  operational  phase  actual  usage  data  are  gathered  for  specific  organiza¬ 
tions  (e.g.,  3M,  VAMOfJC)  by  using  units.  Most  of  these  data  are  reliability, 
maintainability,  availability,  and  cost  data. 

Data  are  also  nniulred  on  tlu*  development  of  tlie  systems  ci f ectiveness 
equal  loss,  for  oporat  tonal  equipment  lli(*se  data  are  probably  in  the.  custody  of 
till'  i'()n  1  pineiit  manaj'.er.  Tyiies  ol  data  needed  are  tlie  figure  of  merit  definitions 
aiul  the  variables  and  equations  used  in  the  availability,  dependability,  and 
eapalilllly  eomputat  Ions.  Since  tlie  sensitivity  of  systems  effectiveness  to 
changes  la  loglstie.s  iaetors  is  to  be  examined,  it  would  he  useful  to  select 
e(iui|siienL H  wliie.h  are  used  on  several  types  of  alreral't  to  determine  the  effect  of 
aircraft  type  on  the  analyses. 

With  the  model,  equipment,  and  data  available,  the  systems  effectiveness 
can  he  computed.  After  the  results  are  validated,  tlie  sensitivity  of  systems 
ef feci. iveueSH  to  the  logistic  factors  can  he  determined  for  the  particular 
equipment.  Ceneral 1 za ti on  of  results  to  other  equipment  and  other  aircraft  can 
be  done  only  after  many  systems  have  been  analyzed.  Once  the  sensitivities  are 


knovm,  resources  con  be  reallocated  to  determine  whether  systems  el: fectlveness 
can  be  improved.  For  instance,  some  of  the  major  causes  of  systems  downtime  are 
shortages  of  and  delays  in  obtaining  spares,  technicians,  technical  data,  tools 
and  test  equipment,  and  training.  Alternatives  can  be  developed,  as  shown  in 
Figure  2,  changing  the  resources  allocated  to  each  of  these  factors  to  determine 
whether  systems  effectiveness  can  be  Improved.  In  this  way  decisions  can  be 
which  can  optimize  the  systems  effectiveness  at  a  specified  resource  level. 

A. 4  recommendations 

When  the  project  had  passed  the  logistic  factors  identification  stage,  and 
the  identification  and  search  for  systems  effectiveness  models  and  data  was  begun, 
it  quickly  became  evident  that  the  level  of  effort  dedicated  to  this  project  was 
not  sufficient  to  accomplish  the  objective  and  Increased  manpower  and  funding 
were  requested.  However,  the  project  sponsor,  considering  tills  to  be  a  high  risk 
urea,  did  not  feel  that  tlie  potential  payback  warranted  committing  additional  „ 
resources  to  the  project,  and  the  project  was  terminated. 


REFERENCES 


1.  Hanifani  D.T.t  "Navy  System  Effect.Weness  Manual,  NAVMAT  P39A1-B 
Proposed,"  Oct  1973,  (AD-769  988). 

2.  CHOP,  A,  "System  and  Cost  Effectiveness  Manual  for  System  Developers," 
Lockheed  Missiles  and  Space  Company,  DO  52S35,  Mr  1970. 

3.  McGowen,  J.J.,  "A  Comparative  Analysis  of  Reliability  Prediction 
Techniques,"  Army  Material  Command,  Texarkana,  Texas,  AP  197!i,  (AD-A009  282). 

A.  NAMAKOFF,  J.,  "Development  of  Reliability  Prediction  Techniques  for 
Non-Avionics,"  Naval  Air  Development  Center,  Warminster,  PA,  Report 
No.  NADC-7308A-50,  Jan  1973. 

5.  James,  L.E.,  et  al.,  "Study  of  Reliability  Prediction  Techniques  For 
Conceptual  Phases  of  Development,"  Hughes  Aircraft  Company,  Fullerton,  CA, 

Oct  1974,  (AD  A-OOl  919). 

6.  "Handbook  cC  Systems  Effectiveness  Moucls,"  Naval  Electronics 
Laboratory  Center,  Sau  Diego-  CA,  Jun  1972,  (AD  7A9  92A). 


INITIAL  DISTRIBUTION 


Cop  It'H 

1  NAVAIH  3A0i;  S.  llurat 
1  Dl.STK 


12  UTU: 


CKNTKR  DISTRIUUTLON 


Cop  It'S 
I 
1 
\ 

1 

10 

10 

1 

1 

1 


Cdilo  Numo 
IHOO  0.11.  OU'iaHiiur 

1H09.3  D.K.  llurrlH 

19  70  M..I.  'Zubko ft' 

1871  11.  ApploRUto 

19  71  1'.  llubui 

3211.1  Hoporta  Dlair IbutUm 

522. 1  Onclaaa  Lib  (C) 

322.2  Uuclaaa  Lib  (A) 

93  1,.  Miiralt 


OTNSRDC  ISSUES  THREE  TYPES  OF  REPORTS 

1.  OTNSRDC  REPORTS,  A  FORMAL  SERIES,  CONTAIN  INFORMATION  OF  PERMANENT  TECH 
NICAL  VALUE.  THEY  CARRY  A  CONSECUTIVE  NUMERICAL  IDENTIFICATION  REGARDLESS  OF 
THEIR  CLASSIFICATION  OR  THE  ORIGINATING  DEPARTMENT. 

2.  DEPARTMENTAL  REPORTS,  A  SEMIFORMAL  SEr'ES,  CONTAIN  INFORMATION  OF  A  PRELIM 
INARY,  TEMPORARY,  OR  PROPRIETARY  NATURE  OR  OF  LIMITED  INTEREST  OR  SIGNIFICANCE, 
THEY  CARRY  A  DEPARTMENTAL  ALPHANUMERICAL  IDENTIFICATION. 

3.  technical  MEMuHANOA,  an  INFORMAL  SERIES,  CONTAIN  TECHNICAL  DOCUMENTATION 
OF  LIMITED  USE  AND  INTEREST.  THEY  ARE  PRIMARILY  WORKING  PAPERS  INTENDED  FOR  IN 
TERNAL  USE.  THEY  CARRY  AN  IDENTIFYING  NUMBER  WHICH  INDICATES  THEIR  TYPE  AND  THE 
NUMERICAL  CODE  OF  THE  ORIGINATING  DEPARTMENT,  ANY  DISTRIBUTION  OUTSIDE  DTNSRDC 
MUST  BE  APPROVED  BY  THE  HEAD  OF  THE  ORIGINATING  DEPARTMENT  ON  A  CASE  BY  CASE 
BASIS. 


